Clustering and classification

Dataset description

The Boston dataset is loaded from the MASS package of R.
This dataset contains Housing values in the suburbs of Boston and has 506 observations and 14 variables, 2 of them are interval and the other ones are numerical.
Description of the dataset can be found here: https://stat.ethz.ch/R-manual/R-devel/library/MASS/html/Boston.html

Variables of the dataset are:
1. ‘crim’ (per capita crime rate by town)
2. ‘zn’ (proportion of residential land zoned for lots over 25,000 sq.ft)
3. ‘indus’ (proportion of non-retail business acres per town)
4. ‘chas’ (Charles River dummy variable (= 1 if tract bounds river; 0 otherwise))
5. ‘nox’ (nitrogen oxides concentration (parts per 10 million))
6. ‘rm’ (average number of rooms per dwelling)
7. ‘age’ (proportion of owner-occupied units built prior to 1940)
8. ‘dis’ (weighted mean of distances to five Boston employment centres)
9. ‘rad’ (index of accessibility to radial highways)
10. ‘tax’ (full-value property-tax rate per $10,000)
11. ‘ptratio’ (pupil-teacher ratio by town)
12. ‘black’ (1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town)
13. ‘lstat’ (lower status of the population (percent))
14. ‘medv’ (median value of owner-occupied homes in $1000s)

library(MASS)
library(dplyr)
## 
## Attaching package: 'dplyr'
## The following object is masked from 'package:MASS':
## 
##     select
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
data(Boston)

# explore the dataset: dimensions, structure and summary
dim(Boston)
## [1] 506  14
str(Boston)
## 'data.frame':    506 obs. of  14 variables:
##  $ crim   : num  0.00632 0.02731 0.02729 0.03237 0.06905 ...
##  $ zn     : num  18 0 0 0 0 0 12.5 12.5 12.5 12.5 ...
##  $ indus  : num  2.31 7.07 7.07 2.18 2.18 2.18 7.87 7.87 7.87 7.87 ...
##  $ chas   : int  0 0 0 0 0 0 0 0 0 0 ...
##  $ nox    : num  0.538 0.469 0.469 0.458 0.458 0.458 0.524 0.524 0.524 0.524 ...
##  $ rm     : num  6.58 6.42 7.18 7 7.15 ...
##  $ age    : num  65.2 78.9 61.1 45.8 54.2 58.7 66.6 96.1 100 85.9 ...
##  $ dis    : num  4.09 4.97 4.97 6.06 6.06 ...
##  $ rad    : int  1 2 2 3 3 3 5 5 5 5 ...
##  $ tax    : num  296 242 242 222 222 222 311 311 311 311 ...
##  $ ptratio: num  15.3 17.8 17.8 18.7 18.7 18.7 15.2 15.2 15.2 15.2 ...
##  $ black  : num  397 397 393 395 397 ...
##  $ lstat  : num  4.98 9.14 4.03 2.94 5.33 ...
##  $ medv   : num  24 21.6 34.7 33.4 36.2 28.7 22.9 27.1 16.5 18.9 ...
summary(Boston)
##       crim                zn             indus            chas        
##  Min.   : 0.00632   Min.   :  0.00   Min.   : 0.46   Min.   :0.00000  
##  1st Qu.: 0.08205   1st Qu.:  0.00   1st Qu.: 5.19   1st Qu.:0.00000  
##  Median : 0.25651   Median :  0.00   Median : 9.69   Median :0.00000  
##  Mean   : 3.61352   Mean   : 11.36   Mean   :11.14   Mean   :0.06917  
##  3rd Qu.: 3.67708   3rd Qu.: 12.50   3rd Qu.:18.10   3rd Qu.:0.00000  
##  Max.   :88.97620   Max.   :100.00   Max.   :27.74   Max.   :1.00000  
##       nox               rm             age              dis        
##  Min.   :0.3850   Min.   :3.561   Min.   :  2.90   Min.   : 1.130  
##  1st Qu.:0.4490   1st Qu.:5.886   1st Qu.: 45.02   1st Qu.: 2.100  
##  Median :0.5380   Median :6.208   Median : 77.50   Median : 3.207  
##  Mean   :0.5547   Mean   :6.285   Mean   : 68.57   Mean   : 3.795  
##  3rd Qu.:0.6240   3rd Qu.:6.623   3rd Qu.: 94.08   3rd Qu.: 5.188  
##  Max.   :0.8710   Max.   :8.780   Max.   :100.00   Max.   :12.127  
##       rad              tax           ptratio          black       
##  Min.   : 1.000   Min.   :187.0   Min.   :12.60   Min.   :  0.32  
##  1st Qu.: 4.000   1st Qu.:279.0   1st Qu.:17.40   1st Qu.:375.38  
##  Median : 5.000   Median :330.0   Median :19.05   Median :391.44  
##  Mean   : 9.549   Mean   :408.2   Mean   :18.46   Mean   :356.67  
##  3rd Qu.:24.000   3rd Qu.:666.0   3rd Qu.:20.20   3rd Qu.:396.23  
##  Max.   :24.000   Max.   :711.0   Max.   :22.00   Max.   :396.90  
##      lstat            medv      
##  Min.   : 1.73   Min.   : 5.00  
##  1st Qu.: 6.95   1st Qu.:17.02  
##  Median :11.36   Median :21.20  
##  Mean   :12.65   Mean   :22.53  
##  3rd Qu.:16.95   3rd Qu.:25.00  
##  Max.   :37.97   Max.   :50.00

Summary shows the min, max, and the first, the second (median), and the third quantum of each variable of the dataset.
The dataset has 506 rows and 14 columns.
The variables have very different ranges and they are not comparable with each other, which probably means that standardization is required before the analysis.

Graphical overview of the dataset:

# plot matrix of the variables
pairs(Boston)

The overview is a bit messy but it offers visual information on how the variables are connected to each other: e.g. there is a hyperbolic relationship between ‘nox’ and ‘dis’, between ‘lstat’ and ‘medv’; almost a linear correlation between ‘rm’ nad ‘lstat’.

library(tidyr)
library(corrplot)
## Warning: package 'corrplot' was built under R version 4.1.2
## corrplot 0.92 loaded
# calculate the correlation matrix and round it
cor_matrix<-cor(Boston) %>% round(digits = 2)

# print the correlation matrix
cor_matrix
##          crim    zn indus  chas   nox    rm   age   dis   rad   tax ptratio
## crim     1.00 -0.20  0.41 -0.06  0.42 -0.22  0.35 -0.38  0.63  0.58    0.29
## zn      -0.20  1.00 -0.53 -0.04 -0.52  0.31 -0.57  0.66 -0.31 -0.31   -0.39
## indus    0.41 -0.53  1.00  0.06  0.76 -0.39  0.64 -0.71  0.60  0.72    0.38
## chas    -0.06 -0.04  0.06  1.00  0.09  0.09  0.09 -0.10 -0.01 -0.04   -0.12
## nox      0.42 -0.52  0.76  0.09  1.00 -0.30  0.73 -0.77  0.61  0.67    0.19
## rm      -0.22  0.31 -0.39  0.09 -0.30  1.00 -0.24  0.21 -0.21 -0.29   -0.36
## age      0.35 -0.57  0.64  0.09  0.73 -0.24  1.00 -0.75  0.46  0.51    0.26
## dis     -0.38  0.66 -0.71 -0.10 -0.77  0.21 -0.75  1.00 -0.49 -0.53   -0.23
## rad      0.63 -0.31  0.60 -0.01  0.61 -0.21  0.46 -0.49  1.00  0.91    0.46
## tax      0.58 -0.31  0.72 -0.04  0.67 -0.29  0.51 -0.53  0.91  1.00    0.46
## ptratio  0.29 -0.39  0.38 -0.12  0.19 -0.36  0.26 -0.23  0.46  0.46    1.00
## black   -0.39  0.18 -0.36  0.05 -0.38  0.13 -0.27  0.29 -0.44 -0.44   -0.18
## lstat    0.46 -0.41  0.60 -0.05  0.59 -0.61  0.60 -0.50  0.49  0.54    0.37
## medv    -0.39  0.36 -0.48  0.18 -0.43  0.70 -0.38  0.25 -0.38 -0.47   -0.51
##         black lstat  medv
## crim    -0.39  0.46 -0.39
## zn       0.18 -0.41  0.36
## indus   -0.36  0.60 -0.48
## chas     0.05 -0.05  0.18
## nox     -0.38  0.59 -0.43
## rm       0.13 -0.61  0.70
## age     -0.27  0.60 -0.38
## dis      0.29 -0.50  0.25
## rad     -0.44  0.49 -0.38
## tax     -0.44  0.54 -0.47
## ptratio -0.18  0.37 -0.51
## black    1.00 -0.37  0.33
## lstat   -0.37  1.00 -0.74
## medv     0.33 -0.74  1.00
# visualize the correlation matrix
corrplot(cor_matrix, method="circle", type = "upper", cl.pos = "b", tl.pos = "d", tl.cex = 0.6)

Red dots in the correlation plot denote negative correlations and blue dots - positive ones. The bigger the circle is, the darker the color of the circle, the stronger the correlation between two variables is.

There is a quite strong correlation between the ‘nox’ parameter (nitrogen oxides concentration)and such parameters as ‘age’ (proportion of owner-occupied units built prior to 1940), ‘dis’ (weighted mean of distances to five Boston employment centres), ‘rad’ (index of accessibility to radial highways), ‘tax’ (full-value property-tax rate per $10,000) and ‘lstat’ (lower status of the population (percent)). The nitrogen oxides concentration is positively correlated with the amount of older buildings, proximity of the highways, higher taxes and population welfare. On the other hand, the more the concentration of nitrogen oxides, the less the weighted mean of distances to employment centers (negative correlation).

Next, there is also a relationship between ‘lstat’ and ‘medv’ (median value of owner-occupied homes in $1000s) variables: the lower the status of the population, the less the median cost of the homes in the area, which can be expected. Same logic can be applied to the ‘lstat’ and ‘medv’ relationships with ‘rm’ (average number of rooms per dwelling): the more rooms in the dwelling, the more the median cost of the homes and the less low-income families can afford this dwelling.

Furthermore, ‘rad’ variable is positively correlated with ‘tax’, which means that more tax is applied to those who live closer to the radial highways.

Lastly, there are rather strong correlations between ‘indus’ (proportion of non-retail business acres per town) and ‘nox’, ‘age’, ‘dis’ and ‘tax’. The more industry there is in the town, the more air pollution it produces, the older are the buildings, the less is the distance to these industrial centers and the higher is the tax.

Based on this analysis, it is safe to say that the variables of the dataset are mostly related to each other and it is possible to build a prediction model using the interplay between parameters.

library(GGally)
## Loading required package: ggplot2
## Registered S3 method overwritten by 'GGally':
##   method from   
##   +.gg   ggplot2
library(ggplot2)
p <- ggpairs(Boston, lower = list(combo = wrap("facethist", bins = 20))) 
p

Only ‘rm’ variable looks like it’s almost normally distributed. Other variables are not distributed normally and have different dimensions.
Therefore, the dataset needs to be scaled.

Dataset standardization

For the reasons stated above we need to scale the dataset first.

# center and standardize variables
boston_scaled <- scale(Boston)

# summaries of the scaled variables
summary(boston_scaled)
##       crim                 zn               indus              chas        
##  Min.   :-0.419367   Min.   :-0.48724   Min.   :-1.5563   Min.   :-0.2723  
##  1st Qu.:-0.410563   1st Qu.:-0.48724   1st Qu.:-0.8668   1st Qu.:-0.2723  
##  Median :-0.390280   Median :-0.48724   Median :-0.2109   Median :-0.2723  
##  Mean   : 0.000000   Mean   : 0.00000   Mean   : 0.0000   Mean   : 0.0000  
##  3rd Qu.: 0.007389   3rd Qu.: 0.04872   3rd Qu.: 1.0150   3rd Qu.:-0.2723  
##  Max.   : 9.924110   Max.   : 3.80047   Max.   : 2.4202   Max.   : 3.6648  
##       nox                rm               age               dis         
##  Min.   :-1.4644   Min.   :-3.8764   Min.   :-2.3331   Min.   :-1.2658  
##  1st Qu.:-0.9121   1st Qu.:-0.5681   1st Qu.:-0.8366   1st Qu.:-0.8049  
##  Median :-0.1441   Median :-0.1084   Median : 0.3171   Median :-0.2790  
##  Mean   : 0.0000   Mean   : 0.0000   Mean   : 0.0000   Mean   : 0.0000  
##  3rd Qu.: 0.5981   3rd Qu.: 0.4823   3rd Qu.: 0.9059   3rd Qu.: 0.6617  
##  Max.   : 2.7296   Max.   : 3.5515   Max.   : 1.1164   Max.   : 3.9566  
##       rad               tax             ptratio            black        
##  Min.   :-0.9819   Min.   :-1.3127   Min.   :-2.7047   Min.   :-3.9033  
##  1st Qu.:-0.6373   1st Qu.:-0.7668   1st Qu.:-0.4876   1st Qu.: 0.2049  
##  Median :-0.5225   Median :-0.4642   Median : 0.2746   Median : 0.3808  
##  Mean   : 0.0000   Mean   : 0.0000   Mean   : 0.0000   Mean   : 0.0000  
##  3rd Qu.: 1.6596   3rd Qu.: 1.5294   3rd Qu.: 0.8058   3rd Qu.: 0.4332  
##  Max.   : 1.6596   Max.   : 1.7964   Max.   : 1.6372   Max.   : 0.4406  
##      lstat              medv        
##  Min.   :-1.5296   Min.   :-1.9063  
##  1st Qu.:-0.7986   1st Qu.:-0.5989  
##  Median :-0.1811   Median :-0.1449  
##  Mean   : 0.0000   Mean   : 0.0000  
##  3rd Qu.: 0.6024   3rd Qu.: 0.2683  
##  Max.   : 3.5453   Max.   : 2.9865
# change the object to data frame so that it will be easier to use the data
boston_scaled <- as.data.frame(boston_scaled)
class(boston_scaled)
## [1] "data.frame"

The scale (min and max) has changed for all the variables. The means of the variables now is zero.

Now we need to create a categorical variable of the crime rate in the Boston dataset (from the scaled crime rate) using quantiles as the break points.

# summary of the scaled crime rate
summary(boston_scaled$crim)
##      Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
## -0.419367 -0.410563 -0.390280  0.000000  0.007389  9.924110

The min value is -0.42 and the max value is 9.92. The 1. quantile is -0.41, the second is -0.39 and the third is 0.007.

# create a quantile vector of crim and print it
bins <- quantile(boston_scaled$crim)
bins
##           0%          25%          50%          75%         100% 
## -0.419366929 -0.410563278 -0.390280295  0.007389247  9.924109610

These are the limits for each category.

# create a categorical variable 'crime'
labels <- c("low", "med_low", "med_high", "high")
crime <- cut(boston_scaled$crim, breaks = bins, include.lowest = TRUE, label=labels)

# look at the table of the new factor crime
table(crime)
## crime
##      low  med_low med_high     high 
##      127      126      126      127

127 values have been fall into the first and the last category, 126 elements fall into the second and the third. Values between -0.419 and -0.411 are in category ‘low’. Values between -0.411 and -0.39 are in category ‘med_low’. Values between -0.39 and 0.00739 are in category ‘med_high’. Values between 0.00739 and 9.92 are in category ‘high’.

# remove original crim from the dataset
boston_scaled <- dplyr::select(boston_scaled, -crim)

# add the new categorical value to scaled data
boston_scaled <- data.frame(boston_scaled, crime)

Here we removed the original variable (crim) from the scaled dataset and added the new categorized variable (crime) to the dataset.

The dataset is prepared now and we can divide the data into training (80%) and testing (20%) sets.

# number of rows in the Boston dataset 
n <- nrow(boston_scaled)

# choose randomly 80% of the rows
ind <- sample(n,  size = n * 0.8)

# create train set
train <- boston_scaled[ind,]
dim(train)
## [1] 404  14
# create test set 
test <- boston_scaled[-ind,]
dim(test)
## [1] 102  14

Train dataset has 404 rows and 14 columns. Test dataset has 102 rows and 14 columns.

Linear Discriminant analysis (LDA)

Let’s train a Linear Discriminant analysis (LDA) classification model. Categorical crime rate is the target variable and all the other variables in the dataset as predictor variables.

lda.fit <- lda(crime ~ ., data = train)
lda.fit
## Call:
## lda(crime ~ ., data = train)
## 
## Prior probabilities of groups:
##      low  med_low med_high     high 
## 0.250000 0.250000 0.259901 0.240099 
## 
## Group means:
##                  zn      indus        chas        nox         rm        age
## low       0.9811956 -0.8869512 -0.07742312 -0.8572715  0.4855809 -0.8516832
## med_low  -0.1229968 -0.2427850 -0.07742312 -0.5467699 -0.1063080 -0.3317107
## med_high -0.3876019  0.1784002  0.10263286  0.3623705  0.1286807  0.4170803
## high     -0.4872402  1.0172187 -0.02879709  1.0706793 -0.3863249  0.8042422
##                 dis        rad        tax     ptratio      black       lstat
## low       0.8336789 -0.6839520 -0.7547741 -0.40660888  0.3767656 -0.76859831
## med_low   0.3273371 -0.5452263 -0.4397746 -0.09562307  0.3123079 -0.12386791
## med_high -0.3372855 -0.3923248 -0.2975691 -0.25572414  0.1010589 -0.04117254
## high     -0.8382266  1.6371072  1.5133254  0.77958792 -0.6491519  0.81561536
##                  medv
## low       0.572916629
## med_low   0.001277373
## med_high  0.178788510
## high     -0.639917042
## 
## Coefficients of linear discriminants:
##                  LD1         LD2         LD3
## zn       0.126902023  0.96825803 -0.92553558
## indus   -0.005449869 -0.23755419  0.17220748
## chas    -0.089113972 -0.04671462  0.05979175
## nox      0.437406027 -0.53939969 -1.42691322
## rm      -0.106661790 -0.11887884 -0.13728521
## age      0.275622541 -0.35075570 -0.32920084
## dis     -0.059965257 -0.44340086  0.11327143
## rad      3.001506088  0.97642592 -0.12644110
## tax     -0.044302064 -0.24463111  0.85351181
## ptratio  0.155924095  0.15200424 -0.38060928
## black   -0.132839662  0.01301554  0.13264485
## lstat    0.161294238 -0.19334485  0.51442561
## medv     0.181956832 -0.28474325 -0.14247864
## 
## Proportion of trace:
##   LD1   LD2   LD3 
## 0.939 0.043 0.018

Prior probabilities of groups: the proportion of training observations in each group. The observations are more or less equally distributed to all the groups (all in the range of 23%-27%, as the numbers change every time we run the analysis and choose randomly the 80% of training data).

Group means denote group center of gravity, the mean of each variable in each group.

Coefficients of linear discriminants are used to form the linear combination of predictor variables that are further used to form the LDA decision rule (LDA provides the coefficient of a linear combination of variables). Proportion of trace is the percentage achieved by each discriminant function.

LD1 seems to be 95.75% whereas the other LDs are not very high, suggesting that the first LDA explains almost all the variability in the dataset.

Next we draw the LDA biplot. The color in the biplot indicates each cluster.

# the function for lda biplot arrows
lda.arrows <- function(x, myscale = 1, arrow_heads = 0.1, color = "orange", tex = 0.75, choices = c(1,2)){
  heads <- coef(x)
  arrows(x0 = 0, y0 = 0, 
         x1 = myscale * heads[,choices[1]], 
         y1 = myscale * heads[,choices[2]], col=color, length = arrow_heads)
  text(myscale * heads[,choices], labels = row.names(heads), 
       cex = tex, col=color, pos=3)
}

# target classes as numeric
classes <- as.numeric(train$crime)

# plot the lda results
plot(lda.fit, dimen = 2, col = classes, pch = classes)
lda.arrows(lda.fit, myscale = 1)

From the plot we can see again that accessibility to radial highways (rad) has the highest LD1 coefficient.

Predicting the test data

To make the prediction of crime rate we will take the crime classes from the test and save them as correct_classes (so that we can compare to it when testing) and remove the crime variable from the test dataset.

# save the correct classes from test data
correct_classes <- test$crime
class(correct_classes)
## [1] "factor"
# remove the crime variable from test data
test <- dplyr::select(test, -crime)
colnames(test)
##  [1] "zn"      "indus"   "chas"    "nox"     "rm"      "age"     "dis"    
##  [8] "rad"     "tax"     "ptratio" "black"   "lstat"   "medv"

There is no longer crime variable in the test dataset.

Next we predict the crime rate and compare the predictions to the correct_classes.

# predict classes with test data
lda.pred <- predict(lda.fit, newdata = test)

# cross tabulate the results
table(correct = correct_classes, predicted = lda.pred$class)
##           predicted
## correct    low med_low med_high high
##   low       13      13        0    0
##   med_low    6      16        3    0
##   med_high   0       6       15    0
##   high       0       0        0   30

The predictions of the model are fairly good, the correct predictions numbers for each category which is situated on the diagonal of the table are the highest numbers across the table.

K-means clustering

Next we load again the Boston dataset and scale it to get comparable distances.

# load the Boston dataset, scale it and create the euclidean distance matrix
library(MASS)
data('Boston')
boston_scaled <- scale(Boston)
boston_scaled <- as.data.frame(boston_scaled)
dist_eu <- dist(boston_scaled, method = "euclidean", diag = FALSE, upper = FALSE, p = 4)
summary(dist_eu)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##  0.1343  3.4625  4.8241  4.9111  6.1863 14.3970

Euclidean is simple the geometric distance between two points, while Manhattan distance observes the absolute differences between the coordinates of two points.

Let’s calculate the manhattan distance.

dist_man <- dist(boston_scaled, method = "manhattan", diag = FALSE, upper = FALSE, p = 4)
summary(dist_man)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##  0.2662  8.4832 12.6090 13.5488 17.7568 48.8618

Next, we run the k-means algorithm on the dataset. K-means clustering algorithm is an unsupervised method, that assigns observations to groups or clusters based on similarity of the objects. K-means needs the number of clusters as an argument, and the optimal number of clusters needs to be defined. First we run K-means clustering using 8 clusters, each identified by a different color. The plot looks very colourful, but is is obvious that the number of clusters is too big.

# k-means clustering
km <-kmeans(Boston, centers = 8)

# plot the Boston dataset with clusters
pairs(Boston, col = km$cluster)

K-means needs the number of clusters as an argument, and the optimal number of clusters needs to be defined. One way to determine the number of clusters is to look at how the total of within cluster sum of squares (WCSS) behaves when the number of cluster changes. The optimal number of clusters is when the total WCSS drops radically.

# MASS, ggplot2 and Boston dataset are available
set.seed(123)

# determine the number of clusters
k_max <- 10

# calculate the total within sum of squares
twcss <- sapply(1:k_max, function(k){kmeans(Boston, k)$tot.withinss})

# visualize the results
library(ggplot2)
qplot(x = 1:k_max, y = twcss, geom = 'line')

It looks like 2 is the optimal number of clusters since the curve changes dramatically on k=2.

Therefore, we will run the k-means analysis with only 2 centroids.

# k-means clustering
km <-kmeans(Boston, centers = 2)

# plot the Boston dataset with clusters
pairs(Boston, col = km$cluster)

So, the optimal number of clusters is 2. More than 2 clusters is abundant. Lets zoom in to have a better look for the analysis.

# zoom in to specific columns
pairs(Boston[1:5], col = km$cluster)

# zoom in to specific columns
pairs(Boston[6:10], col = km$cluster)

# zoom in to specific columns
pairs(Boston[10:14], col = km$cluster)

We can see that clusters denoted by red and black color are distinguishable from each other, which supoorts the idea of having two optimal clusters.

Our previous conclusions based on correlations can be observed here too. The distributions for such pairs as ‘indus’ and ‘nox’, ‘lstat’ and ‘medv’, ‘medv’ and ‘rm’, ‘rm’ and ‘lstat’, ‘dis’ and ‘nox’ prove to be have linear or hyperbolic relationships.

Bonus: k-means on the original Boston data

library(MASS)
data('Boston')
boston_scaled <- scale(Boston)
boston_scaled <- as.data.frame(boston_scaled)
boston_scaled <- dplyr::select(boston_scaled, -crim)
n <- 506
ind <- sample(n,  size = n * 0.8)
ktrain <- boston_scaled[ind,]
ktest <- boston_scaled[-ind,]
km <-kmeans(ktrain, centers = 4)
#length(km)
lda.fit <- lda(km$cluster ~ . , data = ktrain)
lda.fit
## Call:
## lda(km$cluster ~ ., data = ktrain)
## 
## Prior probabilities of groups:
##         1         2         3         4 
## 0.2227723 0.2747525 0.3811881 0.1212871 
## 
## Group means:
##           zn      indus        chas        nox         rm        age        dis
## 1 -0.4812850  0.5190046  0.16512651  0.4636541 -0.5324236  0.6083763 -0.5683231
## 2 -0.4872402  1.0403131 -0.02404347  1.0527448 -0.3806158  0.7760818 -0.8247049
## 3 -0.1241766 -0.6522457  0.06002355 -0.5753066  0.4028566 -0.3996920  0.3579442
## 4  2.2643344 -1.1492006 -0.27232907 -1.1976137  0.6718297 -1.4468053  1.6575639
##          rad        tax     ptratio       black      lstat       medv
## 1 -0.5926685 -0.2923421  0.05697847  0.04711926  0.4630253 -0.4278552
## 2  1.6182167  1.5342238  0.80494617 -0.81318997  0.8336384 -0.7146212
## 3 -0.5523147 -0.7449329 -0.40267404  0.36555371 -0.5852032  0.4973671
## 4 -0.6865511 -0.5704092 -0.76752788  0.34774570 -0.9327205  0.7233699
## 
## Coefficients of linear discriminants:
##                 LD1          LD2         LD3
## zn       0.10622983  1.653335114  1.24563338
## indus    0.47940634 -0.167290290  0.50001998
## chas    -0.01827653 -0.110962089  0.03150472
## nox     -0.06167804 -0.048162595  0.66089368
## rm      -0.05933667  0.159892918 -0.18545570
## age      0.10134636 -0.430198895  0.19061438
## dis     -0.44615071  0.572352328 -0.21526290
## rad      3.58772420  0.718037376 -2.12731486
## tax      1.07739294  0.878746401  1.01263901
## ptratio  0.37148164 -0.070944851  0.57158684
## black   -0.03356943 -0.003761048 -0.08426066
## lstat    0.21415982  0.003520682  0.26131387
## medv    -0.06573643  0.162441226  0.03495705
## 
## Proportion of trace:
##    LD1    LD2    LD3 
## 0.8508 0.1216 0.0276
lda.arrows <- function(x, myscale = 1, arrow_heads = 0.1, color = "red", tex = 0.75, choices = c(1,2)){
  heads <- coef(x)
  arrows(x0 = 0, y0 = 0, 
         x1 = myscale * heads[,choices[1]], 
         y1 = myscale * heads[,choices[2]], col=color, length = arrow_heads)
  text(myscale * heads[,choices], labels = row.names(heads), 
       cex = tex, col=color, pos=3)
}
classes <- as.numeric(train$crime)
plot(lda.fit, dimen = 2, col = classes, pch = classes)
lda.arrows(lda.fit, myscale = 1)

In the plot we can see thebiplot for the LDA analysis using the clusters as target classes. The ‘rad’ variable is the most influencial linear separator for the clusters again. Also, ‘zn’ and ‘tax’ are next most influential variables.

Super-bonus: 3D plots

Next, we create a matrix product, which is a projection of the data points and make a 3D plot of the columns of the matrix product.

model_predictors <- dplyr::select(train, -crime)
# check the dimensions
dim(model_predictors)
## [1] 404  13
dim(lda.fit$scaling)
## [1] 13  3
# matrix multiplication
matrix_product <- as.matrix(model_predictors) %*% lda.fit$scaling
matrix_product <- as.data.frame(matrix_product)

# create 3D plot of the columns of the matrix product 
library(plotly)
## Warning: package 'plotly' was built under R version 4.1.2
## 
## Attaching package: 'plotly'
## The following object is masked from 'package:ggplot2':
## 
##     last_plot
## The following object is masked from 'package:MASS':
## 
##     select
## The following object is masked from 'package:stats':
## 
##     filter
## The following object is masked from 'package:graphics':
## 
##     layout
plot_ly(x = matrix_product$LD1, y = matrix_product$LD2, z = matrix_product$LD3, type= 'scatter3d', mode='markers')

Now we create 3D plot and color it by the crime variable of the test dataset.

# create 3D plot of the columns of the matrix product 
library(plotly)
plot_ly(x = matrix_product$LD1, y = matrix_product$LD2, z = matrix_product$LD3, type= 'scatter3d', mode='markers', color= train$crime)

Finally, we create 3D plot and color it by the clusters of the k-means.

# 3D plot by k means cluster
plot_ly(x = matrix_product$LD1, y = matrix_product$LD2, z = matrix_product$LD3, type= 'scatter3d', mode='markers', color= km$cluster)

The plots above differ only by coloring which highlights specific features. The first plot shows the 3D distribution of the three LDs, colored by the level of crimes. It can be seen that the ‘high’ crime rate is the most defined group that stands further away than most of the points belonging to other categories. The second plot shows the same 3D distribution color coded based on what cluster they belong to. There is no standalone group as in previous plot, the datapoints belong to different clusters without any clear pattern.